Most software is developed as a number of reusable software objects, each designed to carry out one or more tasks. The utility and functionality of the software, as well as the computing system running the software, depend on the proper coding of the source code that is compiled or interpreted for execution by a processor. Coding errors usually cause a deviation from expected functionality of the software and potentially may impact other parts of the computer system (e.g., other applications, databases, the operating system, etc.) Such coding errors not only frustrate the user's computing experience with the software, but can also cause undesired effects throughout the computer system. Therefore, producers of high-quality software expend significant testing and analysis efforts to eliminate errors in their software.
Market demand and the creativity of programmers and designers, however, have led to increasingly complex—yet powerful—software. As the complexity of software increases, so often does the number of lines of source code needed to implement the software. As a direct result, the potential for errors in the source code continues to increase. Accordingly, software must often be tested or analyzed numerous times and in a variety of ways (before and after release) to help ensure that the software operates as intended.
In some analysis environments, a tester develops automated rules or tests (used herein interchangeably) that verify the operation of one or more portions of the software. For example, a rule may automate entering of input field values within a graphical user interface, calling software objects with various input parameters and under a wide range of conditions, gathering the resulting output, and determining if a test passed or failed. A tester (who may or may not be the test developer) can then execute the test case, which provides an indication of whether the object or target code under test passed or failed (and thus whether the object is operating as intended).
Currently, test developers write separate rules for different levels of software verification, i.e., the amount of analysis that each rule performs when determining if an object or piece of code passed or failed. Verification levels for rules that test software vary widely depending on myriad factors. Accordingly, there is generally a tradeoff between the amount of time consumed in running a rule or test case and how thoroughly the software is tested. In particular, the less outputs that are generated and analyzed, the less time consuming the testing becomes. For example, a test developer may write rules for simply testing the stress or load of the software. In such a case, the resulting outputs of the test case may be ignored and the object or targeted code is considered to have passed if the software or system doesn't crash. While this form of analysis allows for a quick test of the software, it does not provide a complete determination of all the effects caused by the software. As such, there is usually much debate and consideration needed in determining the verification level necessary for each rule to appropriately analyze the software.
In order to allow for a wide range of testing verification levels, code analysis tools (e.g., static code analysis tools) have been developed, which provide a semi-automated mechanism for software developers to check their code for problems and inconsistencies. More specifically, test developers populate such tools with a set of rules with various verification levels for analyzing software source, object, or binary code in order to identify the correctness, completeness, and/or quality thereof. These tools typically perform or execute the analysis of the software as a monolithic operation controlled via various forms of configuration. In other words, specific checks or rules applied to the code are either enabled or disabled by source control expressions (such as #pragmas), via tool command-line options, or by settings rendered in a separate control file. Although this allows a tester some control over what types of rules to run at various stages in the development process, there are still several shortcomings and limitations to such approach.
For example, in order to appropriately test code, a tester (i.e., user of the static code analysis tool) needs to have extensive knowledge of the various rules including, but not limited to, at what stages and under what conditions the rules should be applied. Such knowledge of all test cases, however, is typically beyond the skill set of most code developers (since they are typically not specialized as test developers). As such, the tester may not be applying rules at the appropriate times and under the appropriate conditions. Moreover, because such tests are typically static in nature (i.e., they are usually executed by explicit gestures from the tester), there are various time inefficiencies when executing such tests. For example, identical checks may be ran over and over again, issues may be raised at inappropriate junctures in the process (e.g., where they are repeatedly ignored or deferred), and/or errors or faults may not be identified at a sufficiently early time in the development process (i.e., it is axiomatic that the further from check-in a problem is identified or located, the more costly it is to resolve). Accordingly, these and other related shortcomings of typical code analysis tools can often cause a number of problems such as undue work, user and developer frustration, performance problems, unidentified errors or faults, time costs, etc.